A Joint Domain-Specific Pre-Training Method Based on Data Enhancement

State-of-the-art performances for natural language processing tasks are achieved by supervised learning, specifically, fine-tuning pre-trained models such as BERT (Bidirectional Encoder Representation from Transformers). With increasingly accurate models, the size of fine-tuned pre-training corpus is becoming larger and larger. However, very few studies have explored selection corpus. Therefore, this paper proposes a data enhancement-based domain method. At first, task downstream jointly trained to alleviate catastrophic forgetting problem generated existing classical methods. Then, based on hard-to-classify texts identified tasks’ feedback, can be reconstructed selecting similar it. The learning deepen model’s understanding undeterminable text expressions, thus enhancing feature extraction ability texts. Without any pre-processing corpus, experiments conducted two tasks, named entity recognition (NER) classification (CLS). results show that selected proposed method supplement domain-specific information improve performance basic model achieve best compared with other benchmark